Image Processing Apparatus and Image Processing Method

ABSTRACT

A basic pattern generating unit  81  generates a film grain basic pattern  86  in which the histogram of random noise is substantially the same as the film grain basic pattern information  53 . The film grain basic pattern  86  is of (64×64) pixels and an area of (16×16) pixels is cut out and supplied to a multiplier  84 . Strength information  88  is generated based on film grain strength information  54  produced on the encoding side and an average value  87  of a pixel value in the area of (16×16) pixels of decoded image information  42 . The strength of basic pattern  86  is adjusted by this strength information  88 . Film grain image  89  whose strength is adjusted is added to the area of (16×16) pixels of the decoded image information  42  by an adder  85  by the unit of pixel.

TECHNICAL FIELD

The present invention relates to an image processing apparatus and imageprocessing method which are applicable for a case of encoding, forexample, a film image using MPEG-4 AVC (or H.264) encoding.

BACKGROUND ART

A conventional image transmitting/receiving apparatus or imagerecording/reproducing apparatus (hereinafter both will be referred to asjust image transmitting apparatus) will be described with reference toFIG. 1. First, an input image 11 is encoded by an image encoding unit 12and the encoded information is sent or recorded to or in awireless/wired transmission passage 13 or an accumulation medium 14corresponding to a transmission system. Data obtained on areceiving/reproduction side is decoded by an image decoding unit 15 soas to obtain a decoded image 16.

After the amount of the input image data is reduced to one-severalth toone-several tenths by the image encoding unit 12, the data is sent orrecorded. For example, the image data can be lowered to a transmissionrate of 1.5 Mbps by MPEG-4 AVC (or H.264) encoding. Because of suchencoding, a difference between the decoded image 16 and the input image11 is recognized as deterioration by a man.

The conventional image transmitting apparatus reduces the amount of theinput image data at the time of transmission or recording. Generally,the image encoding standard such as the MPEG-4 AVC (or H.264) encodinghas stipulated a method of reducing the data amount effectively withoutremarkable deterioration for the human visual perception. However, undera limited transmission band width or storage capacity, the detailinformation of the image is damaged largely. Particularly, whentransmitting or recording film images such as movie, film graininformation which expresses a large amount of the texture of a movie islost, so that the quality of decoded image is deteriorated remarkably.

FIG. 2A shows an example of film image applied as the input image 11,and FIG. 2B shows an example of the decoded image 16. As evident fromFIGS. 2A and 2B, although the film grain which is detected as thequality of film can be perceived in the input image 21, the decodedimage 22 presents a flat image because most of the film graininformation is lost. Since the film grain is lost, there is a problemthat the film texture cannot be sensed.

Conventionally, Japanese Patent Application Laid-Open (JP-A) No.11-250246 has stated that the sharpness is stressed by suppressingnoises of an image fetched in by a scanner.

The apparatus disclosed in JP-A-11-250246 intends to suppress the grainnoise particular to the film image but not to improve the film imagetexture lost for encoding by high efficiency encoding system on thedecoding side.

Accordingly, an object of the present invention is to provide an imageprocessing apparatus and processing method in which loss of film graininformation is hardly sensed on the decoding side so as to maintain thefilm texture of an input image, thereby consequently improving the imagequality largely.

DISCLOSURE OF THE INVENTION

In order to solve the above problems, a first aspect of the presentinvention provides an image processing apparatus for high efficiencyencoding of an input image signal, including:

area detecting means for detecting an area containing no steep change inan input image;

first extracting means for extracting basic pattern informationindicating a level distribution of pixel values in the area detected bythe area detecting means;

second extracting means for obtaining strength information indicatingthe strength of a film grain to be added; and

means for transmitting or recording encoded data, the basic patterninformation and the strength information. The invention also provides animage processing method on the encoding side to achieve processing ofthe apparatus on the encoding side described above.

A second aspect of the present invention provides an image processingapparatus for receiving or reproducing encoded data by high efficiencyencoding, basic pattern information indicating a level distribution ofpixel values in an area containing no steep change in the input imageand strength information indicating an extent of adding film grain,including:

decoding means for decoding encoded data;

basic pattern generating means for reconstructing a film grain image byconverting a level distribution of random noise to the same one as thelevel distribution indicated in the basic pattern information;

strength adjusting means for adjusting the strength of the film grainimage according to the strength information; and

adding means for adding the film grain image whose strength is adjustedto decoded image information obtained by the decoding means. Theinvention also provides an image processing method on the decoding sideto achieve processing of the apparatus on the decoding side describedabove.

The present invention provides an image processing apparatus for highefficiency encoding an input image signal, transmitting or recordingencoded data and decoding the received or reproduced encoded data,wherein

the encoding side thereof includes:

area detecting means for detecting an area containing no steep change inan input image;

first extracting means for extracting basic pattern informationindicating a level distribution of pixel values in the area detected bythe area detecting means;

second extracting means for obtaining strength information indicatingthe strength of a film grain to be added; and

means for transmitting or recording encoded data, the basic patterninformation and the strength information, and

the decoding side thereof includes:

means for receiving or reproducing the encoded data, the basic patterninformation and the strength information;

basic pattern generating means for reconstructing a film grain image byconverting a level distribution of random noise to the same one as thelevel distribution indicated in the basic pattern information;

strength adjusting means for adjusting the strength of the film grainimage according to the strength information; and

adding means for adding the film grain image whose strength is adjustedto decoded image information obtained by the decoding means. Theinvention also provides an image processing method on the encoding sideand decoding side to achieve processing of the apparatus on the encodingside and decoding side described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of a conventional imagetransmitting apparatus to which the present invention can be applied;

FIGS. 2A and 2B are schematic views showing an example of an input imageand a decoded image in the conventional image transmitting apparatus;

FIG. 3 is a block diagram schematically showing an embodiment of theimage transmitting apparatus of the present invention;

FIG. 4 is a block diagram of an example of a film grain informationextracting unit according to the embodiment of the present invention;

FIG. 5 is a block diagram of an example of a film grain basic patterninformation extracting unit in the film grain extracting unit;

FIG. 6 is a block diagram of an example of a film grain strengthinformation extracting unit in the film grain extracting unit;

FIGS. 7A, 7B and 7C are schematic diagrams for schematically explainingextraction processing of the film grain basic pattern information;

FIG. 8 is a block diagram of an example of a film grain adding unitaccording to the embodiment of the present invention;

FIG. 9 is a block diagram of an example of a basic pattern generatingunit in the film grain adding unit;

FIG. 10 is a schematic diagram for schematically explaining film grainadding processing;

FIG. 11 is a timing chart for explaining a first method of film graininformation extraction and film grain adding processing;

FIG. 12 is a timing chart for explaining a second method of the filmgrain information extraction and film grain adding processing;

FIG. 13 is a timing chart for explaining a third method of the filmgrain information extraction and film grain adding processing;

FIG. 14 is a timing chart for explaining a fourth method of the filmgrain information extraction and film grain adding processing;

FIG. 15 is a timing chart for explaining a fifth method of the filmgrain information extraction and film grain adding processing; and

FIG. 16 is a timing chart for explaining a sixth method of the filmgrain information extraction and film grain adding processing.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, one embodiment of the present invention will be describedwith reference to the accompanying drawings. FIG. 3 shows an imagetransmitting apparatus of the present invention. First, the encodingside will be described. An input image 31 is encoded highly efficientlyby an image encoding unit 32 so as to generate encoded information 34 bycompressing the data amount. The encoded information 34 is sent to awireless/wired transmission passage 36 corresponding to a transmissionsystem or recorded in an accumulation medium 37 such as a digitalversatile disc (DVD).

Reference numeral 33 denotes a film grain information extracting unit.The film grain information extracting unit 33 extracts film graininformation from the input image 31, and the extracted film graininformation 35 is sent to the wireless/wired transmission passage 36corresponding to the transmission system like the encoded information 34or recorded in the accumulation medium 37.

Next, the decoding side will be described. Received encoded information38 is supplied to an image decoding unit 40 on the decoding side, andthe image decoding unit 40 generates decoded image information 42 fromthe received encoded information 38. Reference numeral 41 denotes a filmgrain adding unit. After the film grain adding unit 41 reproduces filmgain information from the received film grain information 39, it addsfilm grain information reconstructed corresponding to the characteristicof the decoded image information 42 and outputs as decoded image 43.

The image encoding unit 32 and the image decoding unit 40 executeprocessing based on the MPEG-4 AVC (or H.264) encoding standard as anexample. In this case, the encoding information 34 has a stream datastructure including units of a predetermined length. The film graininformation 35 is transmitted or recorded using a unit for transmittingSupplemental Enhancement Information (SEI), which is not indispensablefor decoding an animation picture. That is, because the SEI containsinformation which a user defines independently, the film graininformation 35 is transmitted or recorded using the user data.

FIG. 4 shows an example of the film grain extracting unit 33. The filmgrain extracting unit 33 is constituted of a basic pattern extractingunit 51 and a strength extracting unit 52. The basic pattern extractingunit 51 extracts information indicating film grain from the input image31 and outputs as film grain basic pattern information 53. The strengthextracting unit 52 computes information of how much strength is to beadded to a film grain to be added to an area having an average value forevery frame should have, in every range of the pixel value, for example,0-20, 21-40, . . . and its information is outputted as the film grainstrength information 54. The film grain information 35 includes both thefilm grain basic pattern information 53 and the film grain strengthinformation 54.

FIG. 5 shows an example of the basic pattern extracting unit 51. Thebasic pattern extracting unit 51 is composed of a uniform areaextracting unit 61, a filter bank 62 and histogram computing units 63,64, 65, 66 and 67.

The uniform area extracting unit 61 extracts an area of a predeterminedsize (called uniform area) in which film grains are distributeduniformly and an image does not contain a steep change from the inputimage 31 and outputs uniform area data 68. An example of the uniformarea is a flat image like sky image as a background. The uniform areadata 68 is constituted of plural items of pixel data existing in auniform area. In the meantime, to obtain the uniform area data 68, it ispermissible to specify one directly or specify one from candidatesoutputted by the unit.

The uniform area data 68 obtained in this way is divided to a pluralityof sub-bands by filter banks 62 a, 62 b constituted of plural high-passfilters (indicated with H) and low-pass filters (indicated with L).Filters which constitute the filter banks 62 a, 62 b are digital filterswhose filter coefficient is specified two-dimensionally.

In a structure example shown in FIG. 5, each of the filter bands 62 a,62 b is constituted of a high-pass band and a low-pass band.Additionally, a plurality of band pass filters may be provided, and asfor the number of stage, multiple stages thereof may be provided insteadof the two stages shown in FIG. 5.

Outputs of a base band and each sub-band are supplied to the histogramcomputing units 63 to 67 corresponding to each band. The frequencydistribution information (hereinafter referred to as histogram) of thepixel value of each band is extracted by the histogram computing units63 to 67 and the extracted histogram information is outputted as thefilm grain basic pattern information 53.

The histogram information is frequency information indicating thequantity of pixels contained in a predetermined pixel value range(hereinafter referred to as division section). For example, assume thatthere exist four predetermined division sections, namely, −19 to −10, −9to 0, 1 to +10, +11 to +20 and the value of each pixel of the input datainto the histogram computing unit at that time is [−11, −5, −1, +3, +8,+14]. In this case, histogram information outputted is the frequency ofpixel data contained in each range, that is, [1, 2, 2, 1]. The histograminformation is outputted for each band.

FIG. 6 shows an exemplary configuration of the strength extracting unit52. The strength extracting unit 52 includes a uniform area extractingunit 71 and a strength computing unit 72. A uniform area is extractedfrom the input image 31 by the uniform area extracting unit 71 like theuniform area extracting unit 61 and uniform area information 73 isoutputted. In this case, regarding a steep area of the image,information distinguished from other uniform area is outputted as anon-uniform area. As indicated in FIG. 6, a, b, c, d, e and f in theinput image are uniform areas while a filled area is a non-uniform area.

The uniform area information 73 is supplied to the strength computingunit 72. The strength computing unit 72 computes an average pixel valueof the area and a distribution value of the film grain for each uniformarea. To compute the distribution value of the film grain, for example,a method of using the distribution value of the pixel value within theuniform area may be used. Then, the film grain strength information 54is computed using the average pixel value and the distribution value ofthe film grain. The film grain strength information 54 is an averagevalue of the distribution of the film grain in the range of apredetermined area average value.

For example, assuming that the range of an average value of the pixelvalue in a predetermined area, for example, an area of (16×16) pixels is0 to 15, 16 to 31, . . . , 240 to 255 and that a pair of an area averagevalue and a distribution value in the uniform area of the input image is(area average value=28, distribution value=500), (30, 1000), (44, 400),(250, 20), the output film grain strength information is [0,750(=(500+1000)/2, 400, . . . , 20). If the image is constituted ofcomponents R, G, B or Y, Cb and Cr, these are outputted for eachcomponent. The film grain strength information undergoes normalizationprocessing such as turning to 1/1000, so that it is converted to a valuesuitable for being multiplied with a film grain basic pattern generatedon the decoding side.

If the input image is constituted of the component video signal, forexample, Y, Cb and Cr components, a method of detecting the film graininformation with only brightness signal Y and adding film graininformation generated in each of the components Y, Cb and Crcorresponding to a detection result and a method of detecting the filmgrain information for each component and adding the film graininformation generated in each of the components Y, Cb and Crcorresponding to a detection result are available.

The basic pattern extracting processing in the above-described encodingside processing will be described schematically with reference to FIGS.7A to 7C. As shown in FIG. 7A, the uniform area R is detected from theinput image I of a frame. The size of the uniform area R is assumed tobe of a predetermined size, for example, (64×64) pixels. The pixels ofthe uniform area R are divided to a base band, low band and high band bythe filter bank 62 a. Although in the structure of FIG. 5, the low bandis divided depending on the frequency by the filter bank 62 b, FIGS. 7Ato 7C show an example of using only the filter bank 62 a forsimplification of description.

FIG. 7B shows an example of the frequency division. In FIG. 7B, itsabscissa axis indicates spatial horizontal frequency and its ordinateaxis indicates spatial vertical frequency. A rectangular area is baseband B1, an area in which both horizontal and vertical frequencies areraised is high band B2 and an area in which both the horizontal andvertical frequencies are lowered is low band B3. A dotted line indicatespositions of a border of the high band and a solid line indicatespositions of a border of the low band. In this example, the frequencyareas of the low band and high band overlap each other in the vicinityof the middle band.

The output data of each sub-band is supplied to the histogram computingunits 63, 64 and 65. The histogram computing unit 63 computes ahistogram concerning a base band indicated in F1 in FIG. 7C, thehistogram computing unit 63 computes a histogram concerning the baseband indicated in F1 in FIG. 7C, and the histogram computing unit 63computes a histogram concerning a base band indicated in F1 in FIG. 7C.

In FIG. 7C, the ordinate axis indicates the frequency and the abscissaaxis indicates the division section of the pixel value. Each divisionsection has a predetermined range of the pixel value for each band. Thehistograms F1, F2 and F3 indicate film grain basic pattern information.

In FIGS. 7A to 7C, the film grain strength information will not bedescribed. As described above, the film grain strength information is anaverage value of the dispersion of film grains in the range of anaverage value of pixels value in a predetermined area and undergoesnormalization processing as required.

Next, the structure and processing on the decoding side will bedescribed. FIG. 8 shows an example of the film grain adding unit 41 onthe decoding side. The film grain adding unit 41 is composed of a filmgrain basic pattern generating unit 81, a pattern strength computingunit 82, an area average value computing unit 83, a multiplier 84 and anadder 85. The basic pattern generating unit 81 generates a pattern ofthe film grain to be added to the decoded image information 42 from thefilm grain basic pattern information 53, and outputs a film grain basicpattern 86. The pattern of the film grain is assumed to be an area ofthe same size as an area to be set for extracting the basic pattern onthe encoding side, for example, an area of (64×64) pixels.

The area average value computing unit 83 obtains an average value ofpixel values in a predetermined area of the decoded image information42, for example, an area of (16×16) pixels, and outputs an area averagevalue 87. The area average value 87 is supplied to the pattern strengthcomputing unit 82. The pattern strength computing unit 82 determines towhich average section of the film grain strength information 54 the areaaverage value 87 belongs and outputs a film grain strength correspondingto the section as film grain strength information 88.

For example, assuming that the ranges of the average values ofpredetermined areas are 0 to 15, 16 to 31, . . . , 240 to 255 and thefilm grain strength information 54 at that time is [0, 750, 400, . . . ,20], the film grain strength outputted at that time is 750 if theinputted area average value is 23. However, as described above, if thefilm grain strength information is subjected to normalization processingon the encoding side preliminarily, it is converted to a value which canbe used for multiplication for strength adjustment. This normalizationmay be carried out on the decoding side.

The film grain basic pattern 86 is multiplied by the film grain strengthinformation 88 and the multiplier 84. A reconstructed film grain image89 is obtained from the multiplier 84. The film grain image 89 is addedto the decoded image information 42 by the adder 89. The decoded image43 is outputted from the adder 85. In the adder 85, pixels at a positionin which the film grain image 89 corresponds to the (16×16) pixels ofthe decoded image information 42 in the unit of the (16×16) pixels aresummed up. The film grain image 89 is cut out from the basic pattern ofthe (64×64) pixels generated by the basic pattern generating unit 81.

In the meantime, the film grain basic pattern 86 may be added by beingshifted for every area unit to be supplied with the film grain pattern,for example, every 16×16 (pixels) in order to obtain uniformity on animage screen. For example, assuming that the area size is H×V and thepixel value of the pattern is G (h, v) (0≦h<H) (0≦v<V), a patternshifted by a random number may be obtained by generating a random numberrandh, randv (0≦randh<H, 0≦randv<V) and then computing a pattern A to beactually added as followed. This pattern may be assumed to be film grainbasic pattern 86.

A(h,v)=G (mod(h+randh, H), mod(v++randv, V))

FIG. 9 shows an exemplary configuration of the film grain basic patterngenerating unit 81. The basic pattern 81 includes a noise generatingunit 91, histogram matching units 92, 93, 94, 95 and 96, high-passfilters 97 and 99, low-pass filters 98 and 100 and a synthesis filterbank 101.

The noise generating unit 91 generates noise (Gauss noise, random noiseof uniform random numbers) of the same size (H×V) as the film grainbasic pattern, for example, (64×64) pixels. Generated noise is suppliedto the histogram matching unit 92. The histogram matching unit 92converts noise so as to have substantially the same histogram as thefilm grain basic pattern information 53. An example of conversionalgorithm in case where the histogram matching unit 92 is achieved bysoftware processing is shown below.

[Histogram Matching Algorithm in the Histogram Matching Unit]

1. Assume that the minimum value and maximum value of the histogram tobe a matching object in the film grain basic pattern information 53 areCMIN, CMAX and the quantity of division sections of the histogrambetween the CMIN and CMAX is number of bins. bins indicates a divisionsection of the histogram. Further, assume that the cumulativedistribution function of a histogram created, received or reproduced onthe encoding side is CDF ref(bin) (0≦bin<number of bins−1).

2. Assume that the input signal (output of the noise generating unit 91)is val org (h, v) and the cumulative distribution function of the inputsignal is CDF org (bin) (0≦bin<number of bins−1). 3. For each factor ofthe input signal val org(h, v), val repl is obtained according to thefollowing pseudo code and val org (h, v) is replaced with that value.

val=CDF org(((val org(h,v)−CMIN)/(CMAX−CMIN)*(number of bins−1)));

for (i=0; i<number of bins; i++)if (CDF ref(i)>=val)break;if (i==number of bins)i−−;

val repl=i/(number of bins−1)*(CMAX−CMIN)+CMIN;

The other histogram matching units 93, 94, 95 and 96 have a histogrammatching function of the above-described algorithm. An output signal ofthe histogram matching unit 92 is supplied to an analysis high-passfilter 97 and an analysis low-pass filter 98, an output of the analysishigh-pass filter 97 is supplied to the histogram matching unit 93, andan output of the analysis low-pass filter 98 is supplied to thehistogram matching unit 94.

Further, an output signal of the histogram matching unit 94 is suppliedto an analysis high-pass filter 99 and an analysis low-pass filter 100,an output of the analysis high-pass filter 99 is supplied to thehistogram matching unit 95, an output of the analysis low-pass filter100 is supplied to the histogram matching unit 96.

The film grain basic pattern information 53 provides each of thehistogram matching units 92 to 96 with different histogram information.The histogram information F1 in FIG. 7C is supplied to the histogrammatching unit 92 because it concerns the base band. The histograminformation F2 is supplied to the histogram matching unit 93 because itconcerns a high band. The histogram information F3 is supplied to thehistogram matching unit 94 because it concerns a low band.

Division of the sub-band is carried out by the analysis high-passfilters 97 and 99 and the analysis low-pass filters 98 and 100,components of each sub-band are supplied to the synthesis filter bank101. The synthesis filter bank 101 includes a synthesis high-pass filter102 to which an output signal of the histogram matching unit 93 issupplied, a synthesis high-pass filter 103 to which an output signal ofthe histogram matching unit 95 is supplied, and a synthesis low-passfilter 104 to which an output signal of the histogram matching unit 96is supplied.

An output signal of the synthesis high-pass filter 103 and an outputsignal of the synthesis low-pass filter 104 are summed up by an adder105, while an output signal of the synthesis high-pass filter 102 and anoutput signal of the adder 105 are summed up by an adder 106. The filmgrain basic pattern information 86 is fetched out to the output of theadder 106. In the meantime, the structure of the synthesis filter bankcorresponds to the structure of the division of the sub-band on theencoding side.

The strength is corrected or adjusted by multiplying the film grainbasic pattern information 86 with the film grain strength information 88by means of the multiplier 84 as described above. The film grain image89 after its strength is adjusted is added to the decoded imageinformation 42, so that the high quality decoded image 43 having thefilm grain information lost by high efficiency encoding and decoding isobtained.

FIG. 10 explains schematically reconstruction processing of the filmgrain image on the decoding side. The film grain basic pattern 86 whosenoise histogram is substantially the same as the film grain basicpattern information 53 is generated by the basic pattern generating unit81. The film grain basic pattern 86 is of (64×64) pixels and an area of(16×16) pixels is cut out and supplied to the multiplier 84.

The film grain strength information 54 is created on the encoding sideand stored and the strength information 88 is outputted corresponding tothe average value 87 of pixel values in (16×16) pixel area of thedecoded image information 42. The strength of the basic pattern 86 isadjusted by this strength information 88. The film grain image 89 whosestrength is adjusted is added to the area of the (16×16) pixels of thedecoded image information 42 by the adder 85 by the unit of the pixel.

Next, some methods of processing for extraction of the film graininformation and reconstruction of the film grain image will be describedby taking a case of the component video signal as an example. FIG. 11shows a first processing method. According to the first processingmethod, the film grain basic pattern information and film grain strengthinformation are extracted for each of the components Y, Cr and Cb so asto extract these information pieces from each frame. The figure shows animage signal in which scene A and scene B exist. On the decoding side,the film grain is reconstructed using the film grain basic patterninformation and film grain strength information of each frame for eachof the components Y, Cr and Cb.

FIG. 12 shows a second processing method. According to the secondprocessing method, the film grain basic pattern information and the filmgrain strength information are extracted for each of the components Y,Cr and Cb. The film grain basic pattern information is extracted at onlya frame at the head of each scene. On the decoding side, extracted basicpattern information is used commonly for all the frames in the samescene. The film grain strength information is extracted for eachcomponent like the first processing method in each frame.

FIG. 13 shows a third processing method. According to the thirdprocessing method, the film grain basic pattern information is extractedfrom only the brightness signal Y in the component. The film grain basicpattern information is extracted in only a frame at the head of eachscene. On the decoding side, the extracted basic pattern information isused commonly for all frames and all components of the same scene. Thefilm grain strength information is extracted for each component like thefirst processing method in each frame.

FIG. 14 shows a fourth processing method. According to the fourthprocessing method, the film grain basic pattern information and filmgrain strength information are extracted for each of the components Y,Cr and Cb. Both the film grain basic pattern information and film grainstrength information is extracted in only a frame at the head of eachscene. On the decoding side, the extracted basic pattern information andstrength information are used commonly for all the frames in the samescene.

FIG. 15 shows a fifth processing method. According to the fifthprocessing method, the film grain basic pattern information is extractedfrom the brightness signal Y in the components. The film grain basicpattern information is extracted in only a frame at the head of eachscene. On the decoding side, the extracted basic pattern information isused commonly for all the frames and all the components in the samescene. The film grain strength information is extracted for each of thecomponents Y, Cr and Cb. The film grain strength information isextracted in only a frame at the head of each scene. On the decodingside, the extracted film grain strength information is used commonly foreach component in all the frames of the same scene.

FIG. 16 shows a sixth processing method. According to the sixthprocessing method, both the film grain basic pattern information and thefilm grain strength information are extracted from only the brightnesssignal Y in the components. On the decoding side, both the extractedbasic pattern information and strength information are used commonly forall the frames and all the components in the same scene.

Of the above-described first to sixth processing methods, an appropriatemethod is used considering the relation between image qualityimprovement effect by adding the film grains and the quantity of data ofthe film grain information to be transmitted. Further, the first tosixth processing methods may be so constructed that two or more thereofcan be selected and may be switched over by user's operation.

The embodiments of the present invention have been describedspecifically above. The present invention is not restricted to theabove-described embodiments but may be modified in various ways based onthe technical spirit of the invention. For example, as for the divisionmethod of the sub-band, it may be divided to two sections in thehorizontal frequency direction and vertical frequency direction so as toproduce four sub-bands. Further, as a method of processing the noisecomponents according to the film grain basic pattern information, otherthan the above-mentioned algorithm may be adopted. Further, the presentinvention allows use of other encoding method such as MPEG2 than theMPEG-4 AVC (or H.264) as a high efficiency encoding method foranimation.

According to the present invention, the film grain information lost onthe decoding side is extracted efficiently on the encoding side andtransmitted to the decoding side or recorded in the recording medium. Onthe decoding side, the film grain information to be added to an image isreconstructed from the obtained information and added to the decodedimage. Consequently, the decoded image is provided with film texture sothat the image quality can be improved largely.

1. An image processing apparatus for high efficiency encoding of aninput image signal, comprising: area detecting means for detecting anarea containing no steep change in an input image; first extractingmeans for extracting basic pattern information indicating a leveldistribution of pixel values in the area detected by the area detectingmeans; second extracting means for obtaining strength informationindicating the strength of a film grain to be added; and means fortransmitting or recording encoded data, the basic pattern informationand the strength information.
 2. The image processing apparatusaccording to claim 1, wherein the first extracting means is an imageprocessing apparatus employing a frequency distribution of pixel valuesin the area as the basic pattern information.
 3. The image processingapparatus according to claim 1, wherein the first extracting means is animage processing apparatus for dividing the pixels in the area into aplurality of frequency bands and extracting the basic pattern for eachfrequency band.
 4. The image processing apparatus according to claim 1,wherein the second extracting means is an image processing apparatus forgenerating the strength information corresponding to an average value ofthe pixels in each area.
 5. The image processing apparatus according toclaim 1, wherein the first and second extracting means detect the basicpattern information and the strength information for each frame in theinput image.
 6. The image processing apparatus according to claim 1,wherein the first and second extracting means detect the basic patterninformation and the strength information for each scene in the inputimage.
 7. The image processing apparatus according to claim 1, whereinone of the first and second extracting means detects one of the basicpattern information and the strength information for each frame of theinput image while the other of the first and second extracting meansdetects the other of the basic pattern information and the strengthinformation for each scene of the input image.
 8. An image processingapparatus for receiving or reproducing encoded data by high efficiencyencoding, basic pattern information indicating a level distribution ofpixel values in an area containing no steep change in the input imageand strength information indicating an extent of adding film grain,comprising: decoding means for decoding encoded data; basic patterngenerating means for reconstructing a film grain image by converting alevel distribution of random noise to the same one as the leveldistribution indicated in the basic pattern information; strengthadjusting means for adjusting the strength of the film grain imageaccording to the strength information; and adding means for adding thefilm grain image whose strength is adjusted to decoded image informationobtained by the decoding means.
 9. The image processing apparatusaccording to claim 8, wherein an area in the film grain image generatedby the basic pattern generating means is cut out and added to thedecoded image information.
 10. The image processing apparatus accordingto claim 9, wherein the position of the cut out area is changed atrandom.
 11. The image processing apparatus according to claim 8, whereinthe strength adjusting means adjusts the strength in each specified areain the image.
 12. An image processing apparatus for high efficiencyencoding an input image signal, transmitting or recording encoded dataand decoding the received or reproduced encoded data, wherein theencoding side thereof includes: area detecting means for detecting anarea containing no steep change in an input image; first extractingmeans for extracting basic pattern information indicating a leveldistribution of pixel values in the area detected by the area detectingmeans; second extracting means for obtaining strength informationindicating the strength of a film grain to be added; and means fortransmitting or recording encoded data, the basic pattern informationand the strength information, and the decoding side thereof includes:means for receiving or reproducing the encoded data, the basic patterninformation and the strength information; basic pattern generating meansfor reconstructing a film grain image by converting a level distributionof random noise to the same one as the level distribution indicated inthe basic pattern information; strength adjusting means for adjustingthe strength of the film grain image according to the strengthinformation; and adding means for adding the film grain image whosestrength is adjusted to decoded image information obtained by thedecoding means.
 13. An image processing method for high efficiencyencoding an input image signal, comprising: an area detecting step ofdetecting an area containing no steep change in an input image; a firstextracting step of extracting basic pattern information indicating alevel distribution of pixel values in the area detected by the areadetecting step; a second extracting step of obtaining strengthinformation indicating the strength of a film grain to be added; and astep of transmitting or recording encoded data, the basic patterninformation and the strength information.
 14. An image processing methodfor receiving or reproducing encoded data by high efficiency encoding,basic pattern information indicating a level distribution of pixelvalues in an area containing no steep change in an input image, andstrength information indicating an extent of adding a film grain,comprising: a decoding step of decoding encoded data; a basic patterngenerating step of reconstructing a film grain image by converting alevel distribution of random noise to the same one as the leveldistribution indicated in the basic pattern information; a strengthadjusting step of adjusting the strength of the film grain imageaccording to the strength information; and an adding step of adding thefilm grain image whose strength is adjusted to decoded image informationobtained in the decoding step.
 15. An image processing method for highefficiency encoding an input image signal, transmitting or recordingencoded data and decoding the received or reproduced encoded data,wherein processing on the encoding side thereof includes: an areadetecting step of detecting an area containing no steep change in aninput image; a first extracting step of extracting basic patterninformation indicating a level distribution of pixel values in the areadetected by the area detecting step; a second extracting step ofobtaining strength information indicating the strength of a film grainto be added; and a step of transmitting or recording encoded data, thebasic pattern information and the strength information, and processingon the decoding side thereof includes: a step of receiving orreproducing the encoded data, the basic pattern information and thestrength information; a basic pattern generating step of reconstructinga film grain image by converting a level distribution of random noise tothe same one as the level distribution indicated in the basic patterninformation; a strength adjusting step of adjusting the strength of thefilm grain image according to the strength information; and an addingstep of adding the film grain image whose strength is adjusted todecoded image information obtained by the decoding step.